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ABSTRACT 

Multi-output inference tasks, such as multi-label classification, have become increasingly important in recent years. A popular 
method for multi-label classification is classifier chains, in which the predictions of individual classifiers are cascaded along a 
chain, thus taking into account inter-label dependencies and improving the overall performance. Several varieties of classifier 
chain methods have been introduced, and many of them perform very competitively across a wide range of benchmark datasets. 
However, scalability limitations become apparent on larger datasets when modeling a fully-cascaded chain. In particular, the 
methods’ strategies for discovering and modeling a good chain structure constitutes a mayor computational bottleneck. In this 
paper, we present the classifier trellis (CT) method for scalable multi-label classification. We compare CT with several recently 
proposed classifier chain methods to show that it occupies an important niche: it is highly competitive on standard multi-label 
problems, yet it can also scale up to thousands or even tens of thousands of labels. 
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1. INTRODUCTION 

Multi-output classification (MOC) (also known variously as multi-target, multi-objective, and multidimensional classification) 
is the supervised learning problem where an instance is associated with a set of qualitative discrete variables (a.k.a. labels), 
rather than with a single variable]}] Since these label variables are often strongly correlated, modeling the dependencies between 
them allows MOC methods to improve their performance at the expense of an increased computational cost. Multi-label 
classification (MLC) is a special case of MOC where all the labels are binary; it has already attracted a great deal of interest 
and development in machine learning literature over the last few years. In |27l . the authors give a recent review of, and many 
references to, a number of recent and popular methods for MLC. FigurejTjshows the relationship between different classification 
paradigms, according to the number of labels (L = 1 vs. L > 1) and their type (binary [K = 2] or not [A' > 2])- 

There are a vast range of active applications of MLC, including tagging images, categorizing documents, and labelling 
video and other media, and learning the relationship among genes and biological functions. Labels (e.g., tags, categories, 
genres) are either relevant or not. For example, an image may be labelled beach and urban; a news article may be sectioned 
under europe and economy. Relevance is usually indicated by 1, and irrelevance by 0. The general MOC scheme may add 
other information such as month, age, or gender. Note that month £ {1,..., 12} and therefore is not simply irrelevant or not. 
This MOC task has received relatively less attention than MLC (although there is some work emerging, e.g., EH and E3). 
However, most MLC-transformation methods (e.g., treating each label variable as a separate multi-class problem) are equally 
applicable to MOC. Indeed, in this paper we deal with a family of methods based on this approach. Note also that, as any 
integer can be represented in binary form (e.g., 3 £4- [0, 0,0,1,1]), any MOC task can ‘decode’ into a MLC task and vice versa. 

In this paper, we focus on scalable MLC methods, able to effectively deal with large datasets at feasible complexity. Many 
recent MLC methods, particularly those based on classifier chains, tend to be over engineered, investing evermore computational 
power to model label dependencies, but presenting poor scalability properties. In the first part of the paper, we review some 

1 We henceforth try to avoid the use of the term 'class'; it generates confusion since it is used variously in the literature to refer to both the target variable, 

and a value that the variable takes. Rather, we refer to label variables, each of which takes a number of values. 




state-of-the-art methods from the MLC and MOC literature to show that the more powerful solutions are not well-suited to deal 
with large-size label sets. For instance, classifier chains 12 EB consider a full cascade of labels along a chain to model their 
joint probability distribution and, either they explore all possible label orders in the chain, incurring in exponential complexity 
with the number of labels, or they compare a small subset of them chosen at random, which is ineffective for large dimensional 
problems. 

The main contribution of the paper is a novel highly-scalable method: the classifier trellis (CT). Rather than imposing a 
long-range and ultimately computationally complex dependency model, as in classifier chains, CT captures the essential depen¬ 
dencies among labels very efficiently. This is achieved by considering a predefined trellis structure for the underlying graphical 
model, where dependent nodes (labels) are sequentially placed in the structure according to easily-computable probabilistic 
measures. Experimental results across a wide set of datasets show that CT is able to scale up to large sets (namely, thousands 
and tens of thousands of labels) while remaining very competitive on standard MLC problems. In fact, in most of our experi¬ 
ments, CT was very close to the running time of the naive baseline method, which neglects any statistical dependency between 
labels. Also, an ensemble version of CT, where the method is run multiple times with different random seeds and classification 
is done through majority voting, does not significantly outperform the single-shot CT. This demonstrates that our method is 
quite robust against initialization. 

The paper is organized as follows. First, in Section [2] we formalize the notation and describe the problem’s setting. In 
Section [3] we review some state-of-the-art methods from the MLC and MOC literature, as well as their various strategies for 
modeling label dependence. This review is augmented with empirical results. In Section [4] we make use of the studies and 
theory from earlier sections to present the classifier trellis (CT) method. In Section [5] we carry out two sets of experiments: 
firstly we compare CT to some state-of-the-art multi-label methods on an MLC task; and secondly, we show that CT can also 
provide competitive performance on typical structured output prediction task (namely localization via segmentation). Finally, 
in Section[6]we discuss the results and take conclusions. 

Two appendixes have been included to help the readability of the paper and support the presented results. IH0 we compare 
two low-complexity methods to infer the label dependencies from training data. In[B]we review Monte Carlo methods, which 
are required in this paper to perform probabilistic approximate inference of the label set associated to a new test input. 

2. PROBLEM SETUP AND NOTATION 

Following a standard machine learning notation, the n-th feature vector can be represented as 

x ( ") = [x { ™\ ... ,x^ ) ] T e X = Ai x • • • x X D c 

where D is the number of features and X ( i (d = 1..... D) indicates the support of each feature. In the traditional multi-class 
classification task, we have a single target variable which can take one out of K values, i.e., 

y (n) € 3 > = 


and for some test instance x* we wish to predict 

y = h(x*) = argmaxp(j/|x*), (1) 

yey 


K=2 


K>2 



Fig. 1 : Different classification paradigms: L is the number of labels and K is the number of values that each label variable can 
take. 






















K = 2 
L = 3 

yj e {o, 1} 
o—*y = [l,o,o] 
■ —*-y = [o,i,o] 
•—*y = [1,1,0] 

A —>y = [0,0,1] 


Fig. 2: Toy example of multi-label classification (MLC), with K = 2 possible values for each label and L = 3 labels (thus 
Uj £ {0,1} for j = 1,2,3) and, implicitly, D = 2 features. Circles, squares and triangles are elements with only one active 
label (i.e., either y 1 = 1 or yi = 1, but not both). Hexagons show vectors such that t/i = yi = 1. 


in such a way that it coincides with the true (unknown) test label with a high probability]^] Furthermore, the conditional 
distribution p(y |x) is usually unknown and has to be estimated during the classifier construction stage. In the standard setting, 
classification is a supervised task where we have to infer the model h from a set of N labelled examples (training data) 
V = y (n) ) } ;)' r =1 , and then apply it to predict the labels for a set of novel unlabelled examples (test data). This prediction 

phase is usually straightforward in the single-output case, since only one of K values needs to be selected. 

In the multi-output classification (MOC) task, we have L such output labels, 

y^ = [y[ n \...,y[ n) ] 

where 

with Ke £ N+ being the finite number of values associated with the i- th label. For some test instance x*, and provided that we 
know the conditional distribution p(y|x), the MOC solution is given by 

y = h{x*) = argmaxp(y|x*). (2) 

yey 

Once more, p(y|x) is usually unknown and has to be estimated from the training data, V = {(x^ n \ y( n ^)}^ =1 , in order to 
construct the model h. Therein precisely lies the main challenge behind MOC, since h must select one out of T = K L 
possible value 0 clearly a much more difficult task than in Eq. (|T|). Furthermore, finding y for a given x* and p(y|x) is quite 
challenging from a computational point of view for large values of K and Ltmm. 

In MLC, all labels are binary labels, namely K( = 2 for i = 1,..., L, with the two possible label values typically notated 
as ye, £ {0,1} or ye £ {—1, +1}- Figure [ 2 ] shows one toy example of MLC with three labels (thus y £ {0, l} 2 3 ). Because of the 
strong co-occurrence, we can interpret that the first label (y \ = 1) implies the second label (y 2 = 1) with high probability, but 
not the other way around. When learning the model /; (x) in ([2]), the goal of MLC (and MOC in general) is capturing this kind 
of dependence among labels in order to improve classification performance; and to do this efficiently enough to scale up to the 
size of the data in the application domains of interest. This typically means connecting labels (i.e., learning labels together) in 
an appropriate structure. Table[l]summarizes the main notation used in the paper. 

3. BACKGROUND AND RELATED WORK 

In this section, we step through some of the most relevant methods for MLC/MOC recently developed, as well as several works 
specifically related to the novel method, presented in later sections. All the methods discussed here, and also the CT method 
presented in Section [4] aim to build a model for h ,(x) in <[2]) by first selecting a suitable model for the label joint posterior 
distribution p(y|x) and then using this model to provide a prediction y to a new test input x*. It is in the first step where 
state-of-the-art methods present a complexity bottleneck to deal with large sets of labels and where CT offers a significantly 
better complexity-performance trade-off. 


2 Eq.{j} corresponds to the widely used maximum a posteriori (MAP) estimator of y* given x*, but other approaches are possible. 

3 A simplification of K\ x K 2 X • • • x Kl (to keep notation cleaner). 





Table 1: Summary of the main notation. 


Notation 

Description 

X = [®x, ..., x d ] t 

instance / input vector; x G R D 

ye {0,1} 

a label (binary variable) 

y€ K} 

an output (multi-class variable), K possible values 

y = [yi,---,yr] T 

1,-dimensional label/output vector 

®*{(x w ,y w )}L 

Training data set, n = 1, ... ,N 

y = h(x.*) 

binary or multi-class classification (test instance x*) 

y = h(x*) 

multi-label multi-output classification (MLC, MOC) 





(c) Bayesian Classifier Chain (BCC) (d) Conditional Dependency Network (CDN) 

Fig. 3: Several multi-label methods depicted as directed/undirected graphical models. 


3.1. Independent Classifiers 

A naive solution to multi-output learning is training L K'-class models as in Eq. (JTJ, i.e., L independent classifiers (IC)j^]and 
using them to classify L times a test instance x*, as [j/i,..., yi] = [/ii(x*),..., /ii(x*)]. IC is represented by the directed 
graphical model shown in Figure[3](a). Note that this approach implicitly assumes the independence among the target variables, 
i.e., p(y|x) = n£=i p( Z/r|x), which is not the case in most (if not all) multi-output datasets. 

3.2. Classifier Chains 

The classifier chains methodology is based on the decomposition of the conditional probability of the label vector y using the 
product rule of probability: 


L 

p( yl x ) =p(yi\x)Y[p(ye\yi,---,ye-i,x) (3) 

1—2 
L 

~/iwn/<( x >2/ i «---'W- i )> (4) 

e =2 

which is approximated with L probabilistic classifiers, /g(x, yi,..., ye-i). As a graphical model, this approach is illustrated 
by Figure [3] (b). 


4 In the MLC literature, the IC approach is also known as the binary relevance method. 













The complexity associated to learn Eq. 0 increases with L, but with a fast greedy inference, as in f20l . it reduces to 


Ve = axgmaxp(y e \y u .. .,y e _ i,x), 

Vi 


( 5 ) 


for t = 1 ..... L. This is not significant for most datasets, and time complexity is close to that of IC in practice. In fact, it 
would be identical if not for the extra y-\,.... ye — 1 attributes. 

With greedy inference comes the concern of error propagation along the chain, since an incorrect estimate yg will nega¬ 
tively affect all following labels. However, this problem is not always serious, and easily overcome with an ensemble |20l l3l. 
Therefore, although there exist a number of approaches for avoiding error propagation via exhaustive iteration or various search 
options SHI HU, we opt for the ensemble approach. 


3.3. Bayesian Classifier Chains 

Instead of considering a fully parameterized Markov chain model for p(y|x), we can use a simpler Bayesian network. Hence, 
0 becomes 


L 

p(y| x ) = W_v{ye\y pa (e)-,^), (6) 

e=i 

where pa(f) are the parents of the f-th label, as proposed in I25l l26l . known as as Bayesian Classifier Chains (BCC), since it 
may remind us of Bayesian networks. Using a structure makes training the individual classifiers faster, since there are fewer 
inputs to them, and also speeds up any kind of inference. Figure |3](c) shows one example of many possible such network 
structures. 

Unfortunately, finding the optimal structure is NP hard due to an impossibly large search space. Consequently, a recent point 
of interest has been finding a good suboptimal structure, such that Eq. ([6]i can be used. The literature has focused around the idea 
of label dependence (see 0| for an excellent discussion). The least complex approach is to measure marginal label dependence, 
i.e., the relative co-occurrence frequencies of the labels. Such approach has been considered in [253111. I' 1 the latter, the authors 
exploited the frequent sets approach m, which measures the co-occurrence of several labels, to incorporate edges into the 
Bayesian network. However, they noted problems with attributes and negative co-occurrence (i.e., mutual exclusiveness; labels 
whose presence indicate the absence of others). The resulting algorithm (hereafter referred to as the FS algorithm) can deal with 
moderately large datasets, but the final network construction approach ends up being rather involved. 

Finding a graph based on conditional label dependence is inherently more demanding, because the input feature space must 
be taken into account, i.e., classifiers must be trained. Of course, training time is a strongly limiting factor here. However, a 
particularly interesting approach to modelling conditional dependence, the so-called LEAD method, was presented in li26l . This 
scheme tries to remove first the dependency of the labels on the feature set, which is the common parent of all the labels, to 
facilitate learning the label dependencies. In order to do so, LEAD trains first an independent classifier for each label (i.e., it 
builds L independent models, as in the IC approach), and then uses the dependency relations in the residual errors of these 
classifiers to learn a Bayesian network following some standard approach (the errors can in fact be treated exactly as if they 
were labels and plugged, e.g., into the FS approach). LEAD is thus a fast method for finding conditional label dependencies, and 
has shown good performance on small-sized datasets. 

Neither the FS nor the LEAD methods assume any particular constraint on the underlying graph and are well suited for MOC 
in the high dimensional regime because of their low complexity. However, if the underlying directed graph is sparse, the PC 
algorithm and its modifications lfT2l [24I are the state-of-the-art solution in directed structured learning. The PC-algorithm runs 
in the worst case in exponential time (as a function of the number of nodes), but if the true underlying graph is sparse, this 
reduces to a polynomial runtime. However, this is typically not the case in MLC/MOC problems. 

For the sake of comparison between the different MFC/MOC approaches, in this paper we only consider the FS and LEAD 
methods to infer direct dependencies between labels. In order to delve deeper into the issue of structure learning using the FS 
and LEAD methods, in[A]we have generated a synthetic dataset, where the underlying structure is known, and compare their 
solutions and the degree of similarity with respect to the true graphical model. As these experiments illustrate, one of the main 
problems behind learning the graphical model structure from scratch is that we typically get too dense networks, where we 
cannot control the complexity associated to training and evaluating each one of the probabilistic classifiers corresponding to the 
resulting factorization in (|6]». This issue is solved by the classifier trellis method proposed in Section[4] 


3.4. Conditional Dependency Networks (CDN) 

Conditional Dependency Networks represent an alternative approach, in which the conditional distribution p(y|x) factorizes 
according to an undirected graphical model, i.e., 


p(y|x) = 


\ n^g(ygl x )> 

<3=1 


(7) 


where Z is a normalizing constant, 0i( ) is a positive function or potential, and y q is a subset of the labels (a clique in the 
undirected graph). The notion of directionality is dropped, thus simplifying the task of learning the graph structure. Undirected 
graphical models are more natural for domains such as spatial or relational data. Therefore, they are well suited for tasks such 
as image segmentation (e.g., 6). Ml and regular MLC problems (e.g., 0). 

Unlike classifier chain methods, a CDN does not construct an approximation to p(y|x) based on a product of probabilistic 
classifiers. In contrast, for each conditional probability of the form 


p(ye\yi\ 


(t) (t-1) 

Vt-vVt+1 


(t-1) \ 

,y L o x ) 


( 8 ) 


a probabilistic classifier is learnt. In an undirected graph, where all the labels that belong to the same clique are connected to 
each other, it is easy to check that 

p{ye\yi\- ■ ■, ■ ■ •, Vl~ x \ x ) = p(yi\y ne (e), x ), (9) 

where y ne (r) is the set of variables connected to ye in the undirected grapf0 Finally, p(ye\y n e(t), x ) f° r = 1,... ,L is 
approximated by a probabilistic classifier fe(y ne (e), x). 

In order to classify a new test input x*, approximate inference using Gibbs sampling is a viable option. In [B] we present 
the formulation of Monte Carlo approaches (including Gibbs sampling) specially tailored to perform approximate inference in 
MLC/MOC methods based on Bayesian networks and undirected graphical models. 


3.5. Other MLC/MOC Approaches 

A final note on related work: there are many other ‘families’ of methods designed for multi-label, multi-output and structured 
output prediction and classification, including many ‘algorithm adapted’ methods. A fairly complete and recent overview can 
be seen in l27l for example. However, most of these methods suffer from similar challenges as the classifier chains family; and 
similarly attempt to model dependence yet remain tractable by using approximations and some form of randomness 1221 iT8l [23 1. 
To cite a more recent example, ETI uses ‘random graphs’ in a way that resembles 0’s CDN, since it uses undirected graphical 
models, and (25l ’s BCC in the sense of the randomness of the graphs considered. 


3.6. Comparison of State-of-the-art Methods for Extracting Structure 

It is our view that many methods employed for multi-label chain classifiers have not been properly compared in the literature, 
particularly with regard to their method for finding structure. There is not yet any conclusive evidence that modelling the 
marginal dependencies (among Y) is enough, or whether it is advisable to model the conditional dependencies also (for best 
performance); and how much return one gets on a heavy investment in searching for a ‘good’ graph structure, over random 
structures. 

We performed two experiments to get an idea; comparing the following methods: 

IC Labels are independent, no structure. 

ECC Ensemble of 10 CC, each with random order (20) 

EBCC-FS Ensemble of 10 BCCs (25l . based on marginal dependence (8j 

EBCC-LEAD as above, but on conditional dependence, as per (26l 

OCC the optimal CC - of all L\ possible (complete) chain orders 

Results are shown in Table [2] of 5xCV (Cross validation) on two small real-world datasets (Music and Scene); small to 
ensure a comparison with OCC (L\ = 720 possible chain orderings). 


5 In other words, y ne (£j is the so called Markov blanket of ye (3- 



Table 2: Comparison of the classification accuracy of existing methods with 5x CV (Cross validation). We chose the two 
smallest datasets so that the ‘optimal’ OCC could complete. 



IC 

ECC 

OCC 

EBCC-FS 

EBCC-LEAD 

Music 

Scene 

0.517 ± 0.03 
0.595 ±0.02 

0.588 ± 0.03 
0.698 ± 0.01 

0.580 ±0.04 
0.699 ±0.01 

0.582 ± 0.02 
0.644 ±0.02 

0.578 ±0.03 
0.645 ± 0.03 


Figure [4] shows an example of the structure found by BCC-FS and BCC-LEAD in a real dataset, namely Music (emotions 
associated with pieces of music). As a base classifier, we use support vector machines (SVMs), fitted with logistic models (as 
according to OH) in order to obtain a probabilistic output, with the default hyper-parameters provided in the SMO implemen¬ 
tation of the Weka framework m. Table [2] confirms that IC’s assumption of independence harms its performance, and that 
the bulk of the MOC literature is justified in trying to overcome this. However, it also suggests that investing factorial and 
exponential time to find the best-fitting chain order and label combination (respectively) does not guarantee the best results. In 
fact, by comparing the results of ECC with EBCC-LEAD and EBCC-FS, even the relatively higher investment in conditional label 
dependence over marginal dependence (EBCC-LEAD vs EBCC-FS) does not necessarily pay off. Finally, ECC’s performance is 
quite close to MCC. Surprisingly, the ECC method tends to provide excellent performance, even though it only learns randomly 
ordered (albeit fully connected) chains. However, as discussed in Section 3.2 its complexity is prohibitively large in high 
dimensional MLC problems. 



Fig. 4: Graphs derived from the Music dataset, with links based on (a) marginal dependence (FS, label-frequency) and (b) 
conditional dependence (LEAD, error-frequency). Here we have based the links on mutual information, therefore links represent 
both co-occurrences (e.g., quiet —> sad) and mutual exclusiveness (e.g., happy —> sad). Generally, we see that the graph 
makes intuitive sense; amazed and happy are neither strongly similar nor opposite emotions, and thus there is not much 
benefit in modelling them together; same with angry and sad. 


4. A SCALABLE APPROACH: CLASSIFIER TRELLIS (CT) 

The goal for a highly scalable CC-based method brings up the common question: which structure to use. On the one hand, 
ignoring important dependency relations will harm performance on typical MLC problems. On the other hand, assumptions 
must be made to scale up to large scale problems. Even though in certain types of MLC problems there is a clear notion of 
the local structure underlying the labels (e.g., in image segmentation pixels next to each other should exhibit interdependence), 
this assumption is not valid for general MLC problems, where the 4th and 27th labels might be highly correlated for example. 
Therefore, we cannot escape the need to discover structure, but we must do it efficiently. Furthermore, the structure used should 
allow for fast inference. 

Our proposed solution is the classifier trellis (CT). To relieve the burden of specification ‘from scratch’, we maintain a fixed 
structure, namely, a lattice or trellis (hence the name). This escapes the high complexity of a complete structure learning (as in 

















Fig. 5 : Three possible directed trellises for L = 9. Each trellis is defined by a fixed pattern for the parents of each vertex 
(varying only near the edges, where parents are not possible). Note that no directed loops can exist. 


GOi and J3j), and at the same time avoids the complexity involved in discovering a structure (e.g., IBS and ll26lD . Instead, we a 
impose a structure a-priori, and only seek an improvement to the order of labels within that structure. 

Figure [5] gives three simple example trellises for L = 9 (specifically, we use the first one in experiments). Each of the 
vertices £ £ {1,..., L} of the trellis corresponds to one of the labels of the dataset. Note the relationship to Eq. (Jhji; we simply 
fix the same pattern to each pa(£). Namely, the parents of each label are the labels laying on the vertices above and to the left 
in the trellis structure (except, obviously, the vertices at the top and left of the trellis which form the border). A more linked 
structure will model dependence among more labels. 

Hence, instead of trying to solve the NP hard structure discovery problem, we use a simple heuristic (label-frequency- 
based pairwise mutual information) to place the labels into a fixed structure (the trellis) in a sensible order: one that tries to 
maximize label dependence between parents and children. This ensures a good structure, which captures some of the main label 
dependencies, while maintaining scalability to a large number of labels and data. Namely, we employ an efficient hill climbing 
method to insert nodes into this trellis according to marginal dependence information, in a manner similar to the FS method in 

m. 

The process followed is outlined in Algorithm^ to which we pass a pairwise matrix of mutual information, where 


I(Y e -Y k ) 


E E p(ye,Vk) log 

yt^yt yk&yk 


f p(ye,yk) \ 
\p(yt)p(.yk))' 


Essentially, new nodes are progressively added to the graph based on the mutual information. Since the algorithm starts by 
placing a random label in the upper left corner vertex, a different trellis will be discovered for different random seeds. Each 
label ijk is directly connected to a fixed number of parent labels in the directed graph (e.g., in Figure[5]each node has two parents 
in the first two graphs, and four in the third one) - except for the border cases where no parents are possible. The computational 
cost of this algorithm is 0(L 2 ), but due to the simple calculations involved, in practice it is easily able to scale up to tens of 
thousands of labels. Indeed, we show later in Section [5]that this method is fast and effective. Furthermore, it is possible to limit 
the complexity by searching for some number < L of labels (e.g., building clusters of labels). 

Given a proper user-defined parent pattern pa{£) (see Figure [5]», we are ensured that the trellis obtained in Algorithm [T] is a 
directed acyclic graph. Hence, there is no need to check for cycles during construction, which is a very time consuming stage 
in many algorithms (e.g., in the PC algorithm, see Section [3G| . We can now employ probabilistic classifiers to construct an 
approximation to p(y|x) according to the directed graph. This approach is simply referred to as the Classifier Trellis (CT). 
Afterwards, we can either do inference greedily or via Monte Carlo sampling (see[B]for a detailed discussion of Monte Carlo 
methods). 

Alternatively, note that we can interpret the trellis structure provided by Algorithm |T] in terms of an undirected graph, 
following the approach of Classifier Dependency Networks, described in Section |3.4| For example, in Figure [5] (middle), we 
would get (with directionality removed) ne(5) = {2,4, 6,8}. This compare^ to pa(5) = {1,2,4}. We refer to this approach 
as the Classifier Dependency Trellis (CDT). 

Both CT and CDT are outlined in Algorithm [2] and Algorithm [3] respectively. Some may argue that the undirected version 
is more powerful, since learning an undirected graph is typically easier than learning a directed graph that encodes causal 
relations. However, CDTconstructs an undirected graphical model where greedy inference cannot be implemented and we have 
to rely on (slower) Monte Carlo sampling methods in the test stage. This effect can be clearly noticed in Table[8] 


6 We did not add the diagonals in the set ne(5), since we wish that it be comparable in terms of the number of connections 








































Algorithm 1 Constructing a Classifier Trellis 


input : Y (an L x N matrix of labels), W (width, default: \JL), a parent-node function pa(-) 

begin 

I learnDependencyMatrix(Y') ; 

S = SHUFFLE({1, ..., L}) ; 
si = Si; 

for l = 2,..., L; do 

S = S\a t -i\ 

se = argmax fceS £ jepaW /(%; 2/fc); 

end 

end 

Si S 2 • • • Siy-l SVK 


output: the trellis structure. 


Sl-W Sl-W+1 • • • SL-1 Sl 


We henceforth assume that yg = y S( ,. 


Algorithm 2 Classifier Trellis (CT) 

Train (V) begin 

Find the directed trellis graph using Algorithm[I] 

Train classifiers fi,..., /^, each taking all parents in the directed graph as additional features, such that 

/f(x) «p(2/£|2/pa(£),x). 

end 

Test(x*) begin 

for i = {si,. .., s_l}; do 

yg = argmax^ p(^|y pa (*), x*) 

end 

return y = [yi, ■ ■. , Vl] 

end 


Finally, we will consider a simple ensemble method for CT, similar to that proposed in ED or l25l to improve classifier 
chain methods: M CT classifiers, each built from a different random seed, where the final label decision is made by majority 
voting. This makes the training time and inference M times larger. We denote this method as ECT. A similar approach could be 
followed for the CDT (thus building an ECDT), but, given the higher computational cost of CDT during the test stage, we have 
concerns regarding the scalability of this approach, so we have excluded it from the simulations. 


5. EXPERIMENTS 


Firstly, in Section 5.1 we compare E/CT and CDT with some high-performance MLC methods (namely ECC, BCC, MCC) that 


were discussed in Section [3] We show that an imposed trellis structure can compete with fully-cascaded chains such as ECC 
and MCC, and discovered structures like those provided by BCC. Our approach based on trellis structures achieves similar MLC 
performance (or better in many cases) while presenting improved scalable properties and, consequently, significantly lower 
running times. 

All the methods considered are listed in Table [3] In Table [4] we summarize their complexity. DN represents the input 
dimensions (the dataset-dependent number of features times number of instances); we use M = 10 ensemble methods and 
T = 100 Gibbs iterations for CDT. While this complexity is just an intuitive measure, the experimental results reported in 
Section |5TT| confirm that CT running times are indeed very close to IC. 

Table [5] summarizes the collection of datasets that we use, of varied type and dimensions; most of them familiar to the 
MLC/MOC community I22ll3ll20l. 

The last two sets (Local400 and LocallOk) are synthetically generated and they correspond to the localization problem 
















Algorithm 3 Classifier Dependency Trellis (CDT) 

Train (V) begin 

Find the undirected trellis graph using Algorithm[T] 

Train classifiers fi,..., each taking all neighbouring labels as additional features, such that 

/r(x) ^p{yt |y ne (f),x). 

end 

Test(x*) begin 

for t = 1,..., T c ,..., T; do 

for £ = SHUFFLE{1,..., L}; do 

V( ) ~K»|yneW,X*) 

end 

end 

return y = ^ [ E Tc <t<T Vi ] . • ■ •. Er c <t<r vf] 

end 

where T r , T : settling time (discarded samples) and total iterations. 


Table 3: Methods tested in experiments. 


Key Description/Name 


Reference 


IC Independent Classifiers 

ECC Ensemble of 10 random CCs (majority vote per label) 

MCC Best of 10 random CC 

EBCC Ensemble of min(10, L) BCCs (discovered directed graph) 

CT Classifier trellis, as in Figure|5j left (directed trellis) 

ECT Ensemble of 10 CTs 

CDT Classifier dependency trellis, as in Figure[5] middle (but undirected ); T = 100, T c = 10 


Sec. J_ 

Sec. 3 f20l 
Sec. 3 El 
Sec. 3 [25] 
Sec. 3 Alg. 
Sec. 3 
Sec. 4 Alg. 
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I 














Table 4: Complexity per algorithm, roughly sorted by training complexity. DN represents the input dimensions (number of 
features times number of instances). Train complexity indicates roughly how many values are looked at by a classifier. Test 
complexity indicates how many (times) individual models are addressed as inference. 


method 

train complexity 

test complexity 

IC 

O(LDN) 

O(L) 

CT 

O(LDN) 

O(L) 

EBCC 

0{M x LDN) 

O(ML) 

CDT 

O(LDN) 

o\tl) 

ECT 

0(M x LDN) 

0(ML) 

ECC 

Q(M x L[D + L\N) 

O(ML) 


Table 5: A collection of datasets and associated statistics, where LC is label cardinality: the average number of labels relevant 
to each example. 



N 

L 

D 

LC 

Type 

Music 

593 

6 

72 

1.87 

audio 

Scene 

2407 

6 

294 

1.07 

image 

Yeast 

2417 

14 

103 

4.24 

biology 

Medical 

978 

45 

1449 

1.25 

medical/text 

Enron 

1702 

53 

1001 

3.38 

text 

TMC2007 

28596 

22 

500 

2.16 

text 

MediaMill 

43907 

101 

120 

4.38 

video 

Delicious 

16105 

983 

500 

19.02 

text 

Local400 

10000 

400 

30 

12.84 

localization 

Local 10k 

50000 

10000 

30 

25.78 

localization 


described in Section [5~2l 

We use some standard metrics from the multi-label literature (see, e.g., US), namely, 


iy J-j 

Hamming Score — NL E£[^=r]> 

n =1 j —1 


Exact Match := — [y (n) = y (n) ], 

n—1 


Accuracy 7 


1 * |y(") A y(»)| 

N |y(") V y(n) | ’ 


where [A] is an identity function, returning 1 if condition A is true, whereas A and V are the bitwise logical AND and OR 
operations, respectively. 


5.1. Comparison of E/CT to other MLC methods 

First of all, to confirm that our proposed hill climbing strategy can actually have a beneficial effect (i.e., an improvement over a 
random trellis), we do a 10 x 10 cross validation (CV) on the smaller datasets. Results are displayed in Table [6] A significant 
increase in performance can be seen, with a decrease in the standard deviation (especially relevant in the Scene dataset). This 
confirms that the proposed hill climbing strategy can help in optimizing performance and decreasing the sensitivity of CT wrt 
the initialization. 


7 Known commonly as Jaccard Index in information retrieval circles. 













Table 6: Comparing CT accuracy (on Scene, Music) without (just a random trellis) and with our ‘hill-climbing’ (HC) method, 
over 10 x 10 CV. 



CT-random 

CT-HC 

Music 

0.536 ± 0.042 

0.541 ±0.031 

Scene 

0.639 ±0.210 

0.678 ±0.028 


Then, we compare all the methods listed in Table[3]on all the datasets of Table[5] The results of predictive performance are 
displayed in Table[7] and running times can be seen in Table[8] On the small datasets (L < 100) we use Support Vector Machines 
as base classifiers, with fitted logistic models (as in mi) for CDN (which requires probabilistic output for inference). As an 
alternative, other authors have used Logistic Regression directly (e.g., 00) due to its probabilistic output. In our experience, 
we obtain better and faster all-round performance with SVMs. Note that, for best accuracy, it is highly recommended to tune 
the base classifier. However, we wish to avoid this “dimension” and instead focus on the multi-label methods. On the larger 
datasets (where L > 100) we instead use Stochastic Gradient Descent (SGD), with a maximum of only 100 epochs, to deal with 
the scale presented by these large problems. All our methods are implemented and made available within the Meka framework 0 
an open-source multi-output learning framework based on the Weka machine learning framework ifTOl . The SMO SVM and 
SGD implementations pertain to Weka. 

Results confirm that both ECT and CT are very competitive in terms of performance and running time. Using the Hamming 
score as figure of merit and given the running times reported, CT is clearly superior to the rest of methods. Note that Table [8] 
shows that CT’s running times are close to IC, namely the method that neglects all statistical dependency between labels. With 
respect to exact match and accuracy performance results, which are measures oriented to the recovery of the whole set of labels, 
ECT and CT achieve very competitive results with respect to ECC and MCC, which are high-complexity methods that model 
the full-chain of labels. Table [8] reports training and test average times computed for the different MLC methods. We also 
include explicitly the number L of labels per dataset. Note also that even though ECT considers a set of 10 possible random 
initializations, it does not significantly improve the performance of CT (a single initialization) for most cases, which suggest that 
the hill climbing strategy makes the CT algorithm quite robust with respect to initialization. Regarding scalability, note that the 
computed train/running times for CT scale roughly linearly with the number of labels L. For instance, in the MediaMill dataset 
L = 101 while in Delicious L is approximately one order of magnitude higher, L = 983. As we can observe, CT running times 
are multiplied approximately by a factor of 10 between both datasets. The same conclusions can be drawn also for the largest 
datasets, compare for instance running times between LocaMOO and LocallOk. Finally, CDT, our scalable modification of CDN, 
shows worse performance than CT while it requires larger test running times. 

In order to illustrate the most significant statistical differences between all methods, in Figure [6] we include the results of 
the Nemenyi test based on Table [7] and Table [8] However, note that here we excluded the two rows with DNFs. The Nemenyi 
test f6j rejects the null hypothesis if the average rank difference is greater than the critical distance q v yjN a(Na ± l)/6Nr> 
over Na algorithms and Nr> datasets, and q p according to the q table for some p value (we use p = 0.90). Any method with a 
rank greater than another method by at least the critical distance, is considered statistically better. In Figure[6] for each method, 
we place a bar spanning from the average rank of the method, to this point plus the critical distance. Thus, any pair of bars that 
do not overlap correspond to methods that are statistically different in terms of performance. Note that, regarding both training 
and test running times, CT overlaps considerably with IC, whereas other methods such as ECC and MCC need significantly 
more training time. We can say that ECC and ECT are statistically stronger than IC and CDT, but not so wrt exact match. CT 
performs particularly well on the Hamming score, indicating that error propagation is limited, compared to other CC methods. 

In the following section we present the framework behind the localization datasets Local400 and Local 1 0k in the tables 
presented above. 


f http://meka.sourceforge.net 








Table 7: Predictive performance and dataset-wise (rank). DNF = Did Not Finish (within 24 hours or 2 GB memory). 


Accuracy 


Dataset 

IC 

ECC 

MCC 

EBCC 

CT 

ECT 

CDT 

Music 

0.483 (7) 

0.572 (2) 

0.568 (4) 

0.566 (5) 

0.577 (1) 

0.571 (3) 

0.505 (6) 

Scene 

0.571 (7) 

0.684 (2) 

0.685 (1) 

0.618(4) 

0.602 (6) 

0.666 (3) 

0.604 (5) 

Yeast 

0.502 (6) 

0.538 (2) 

0.534 (4) 

0.535 (3) 

0.533 (5) 

0.541 (1) 

0.438 (7) 

Medical 

0.699 (7) 

0.733 (3) 

0.721 (5) 

0.731 (4) 

0.755 (2) 

0.769 (1) 

0.704 (6) 

Enron 

0.406 (5) 

0.448 (1) 

0.403 (6) 

0.441 (3) 

0.409 (4) 

0.443 (2) 

0.310(7) 

TMC07 

0.614 (5) 

0.645 (1) 

0.619(4) 

0.628 (3) 

0.613 (6) 

0.633 (2) 

0.601 (7) 

MediaMill 

0.379 (2) 

0.350 (5) 

0.349 (6) 

0.375 (3) 

0.391 (1) 

0.344 (7) 

0.374 (4) 

Delicious 

0.122 (3) 

DNF 

0.121 (5) 

DNF 

0.127 (2) 

0.157(1) 

0.122 (3) 

Local400 

0.536 (7) 

0.625 (1) 

0.583 (3) 

0.578 (4) 

0.542 (6) 

0.587 (2) 

0.559 (5) 

Local 10k 

0.125 (4) 

DNF 

DNF 

0.175 (1) 

0.133 (3) 

0.166 (2) 

0.122 (5) 

avg rank 

5.30 

2.12 

4.22 

3.33 

3.60 

2.40 

5.50 

Hamming Score 

Dataset 

IC 

ECC 

MCC 

EBCC 

CT 

ECT 

CDT 

Music 

0.785 (6) 

0.795 (3) 

0.789 (5) 

0.800 (1) 

0.798 (2) 

0.795 (3) 

0.768 (7) 

Scene 

0.886 (4) 

0.892 (1) 

0.892(1) 

0.886 (4) 

0.884 (6) 

0.891 (3) 

0.871 (7) 

Yeast 

0.800(11 

0.789 (4) 

0.794 (2) 

0.787 (5) 

0.791 (3) 

0.786 (6) 

0.719(7) 

Medical 

0.988 (4) 

0.988 (4) 

0.989 (2) 

0.988 (4) 

0.990(1) 

0.989 (2) 

0.986 (7) 

Enron 

0.943 (1) 

0.940 (4) 

0.942 (3) 

0.939 (5) 

0.943 (1) 

0.939 (5) 

0.922 (7) 

TMC07 

0.947 (2) 

0.948 (1) 

0.947 (2) 

0.946 (5) 

0.947 (2) 

0.946 (5) 

0.937 (7) 

MediaMill 

0.965 (2) 

0.947 (7) 

0.958 (4) 

0.954 (5) 

0.966(1) 

0.951 (6) 

0.965 (2) 

Delicious 

0.982(1) 

DNF 

0.981 (4) 

DNF 

0.982(1) 

0.981 (4) 

0.982(1) 

Local400 

0.968 (3) 

0.969 (1) 

0.967 (6) 

0.968 (3) 

0.969(1) 

0.968 (3) 

0.962 (7) 

Local 10k 

0.968 (1) 

DNF 

DNF 

0.968 (1) 

0.968 (1) 

0.968 (1) 

0.968(1) 

avg rank 

2.50 

3.12 

3.22 

3.67 

1.90 

3.80 

5.30 

Exact Match 

Dataset 

IC 

ECC 

MCC 

EBCC 

CT 

ECT 

CDT 

Music 

0.252 (7) 

0.327 (1) 

0.292 (5) 

0.302 (4) 

0.312(2) 

0.312(2) 

0.257 (6) 

Scene 

0.491 (7) 

0.579 (2) 

0.638 (1) 

0.516(5) 

0.542 (4) 

0.557 (3) 

0.503 (6) 

Yeast 

0.160 (5) 

0.190(3) 

0.212(1) 

0.150 (6) 

0.198 (2) 

0.169 (4) 

0.067 (7) 

Medical 

0.614 (4) 

0.612(5) 

0.634 (3) 

0.612(5) 

0.670(1) 

0.655 (2) 

0.598 (7) 

Enron 

0.121 (3) 

0.112(5) 

0.126(1) 

0.114(4) 

0.123 (2) 

0.112(5) 

0.067 (7) 

TMC07 

0.330 (4) 

0.342 (2) 

0.345 (1) 

0.316(6) 

0.341 (3) 

0.317 (5) 

0.263 (7) 

MediaMill 

0.055 (2) 

0.034 (5) 

0.053 (3) 

0.019 (6) 

0.058 (1) 

0.007 (7) 

0.052 (4) 

Delicious 

0.003 (3) 

DNF 

0.006(1) 

DNF 

0.004 (2) 

0.002 (5) 

0.003 (3) 

Local400 

0.064 (4) 

0.108(2) 

0.129(1) 

0.059 (6) 

0.079 (3) 

0.063 (5) 

0.029 (7) 

Local 10k 

0.000 (1) 

DNF 

DNF 

0.000(1) 

0.000(1) 

0.000 (1) 

0.000 (1) 

avg rank 

4.00 

3.12 

1.89 

4.78 

2.10 

3.90 

5.50 



Table 8: Time results (seconds). DNF = Did Not Finish (within 24 hours or 2 GB memory). 


Training Time 


Dataset 

L 

IC 

ECC 

MCC 

EBCC 

CT 

ECT 

CDT 

Music 

6 

1(3) 

4(6) 

4(7) 

2(5) 

0(1) 

2(4) 

1(2) 

Scene 

6 

3(1) 

10(5) 

28(7) 

7(4) 

3(3) 

10(6) 

3(2) 

Yeast 

14 

11(3) 

53 (6) 

79 (7) 

45 (5) 

5(2) 

26 (4) 

5(1) 

Medical 

45 

4(2) 

19(6) 

67 (7) 

17 (4) 

3(1) 

19(5) 

5(3) 

Enron 

53 

51 (3) 

207 (6) 

734 (7) 

95 (4) 

24(1) 

100 (5) 

37 (2) 

TMC07 

22 

11402 (2) 

48019(6) 

73433 (7) 

34559 (4) 

10847 (1) 

44986 (5) 

13547 (3) 

MediaMill 

101 

42(1) 

347 (6) 

1121(7) 

238 (5) 

45 (2) 

219 (4) 

55 (3) 

Delicious 

983 

468(1) 

DNF 

18632 (5) 

DNF 

529 (2) 

2791(4) 

599 (3) 

Local400 

400 

2(1) 

15(5) 

57(7) 

8(3) 

2(2) 

9(4) 

31 (6) 

Local 10k 

10 4 

3(1) 

DNF 

DNF 

13 (4) 

3(2) 

15(5) 

3(3) 

avg rank 


1.80 

5.75 

6.78 

4.22 

1.70 

4.60 

2.80 


Test Time 


Dataset 

L 

IC 

ECC 

MCC 

EBCC 

CT 

ECT 

CDT 

Music 

6 

0(3) 

1 (7) 

0(1) 

0(5) 

0(4) 

0(2) 

0(6) 

Scene 

6 

0(1) 

1 (5) 

0(2) 

0(4) 

0(3) 

2(6) 

7(7) 

Yeast 

14 

1(2) 

3(5) 

3(6) 

1(3) 

0(1) 

2(4) 

8(7) 

Medical 

45 

4(2) 

28 (6) 

9(3) 

26 (5) 

2(1) 

22 (4) 

141 (7) 

Enron 

53 

4(1) 

112(6) 

8(3) 

19(4) 

4(2) 

45 (5) 

310(7) 

TMC07 

22 

7(2) 

50 (5) 

3(1) 

42 (4) 

7(3) 

76 (6) 

534 (7) 

MediaMill 

101 

15(1) 

211 (5) 

31(2) 

143 (4) 

32(3) 

972 (6) 

5469 (7) 

Delicious 

983 

167 (1) 

DNF 

322 (3) 

DNF 

207 (2) 

7532(4) 

31985 (5) 

Local400 

400 

1(1) 

17 (6) 

3(3) 

9(4) 

1 (2) 

17(5) 

398 (7) 

Local 10k 

10 4 

4(2) 

DNF 

DNF 

18(3) 

4(1) 

39 (4) 

4228 (5) 

avg rank 


1.60 

5.62 

2.67 

4.00 

2.20 

4.60 

6.50 


5.2. A Structured Output Prediction Problem 

We investigate the application of CT (and the other MLC methods) to a type of structured output prediction problem: segmenta¬ 
tion for localization. In this section we consider a localization application using light sensors, based on the real-world scenario 
described in fl6l , where a number of light sensors are arranged around a room for the purpose of detecting the location of a 
person. We take a ‘segmentation’ view of this problem, and use synthetic models (which are based on real sensor data) to gen¬ 
erate our own observations, thus creating a semi-synthetic dataset, which allows us to easily control the scale and complexity. 
Figure [7] shows the scenario. It is a top-down view of a room with light sensors arranged around the edges, one light source (a 
window, drawn as a thin rectangle) on the bottom edge and four targets. Note that targets can only be detected if they come 
between a light sensor and the light source, so the target in the lower right corner is undetectable. 

We divide the scenario into L = W x W square ‘tiles’, representing Y..... Y^. Given an instance n, 


y ' = 


( n ) 
1/1,1 

(n) 

1/1,2 ■ ■ ■ 

(n) 

■ ■ ■ Vi.w 

( n ) 

Vw, 1 

(n) 

Vw.i ■ ■ ■ 

( n ) 

■ ■ ■ Vw,w_ 


where y l -"' ) = 1 if the i, j -th tile (i.e., pixel) is active, and y^ n ) = 0 otherwise, with i,j £ 


have binary sensor observations = [x\ n ' > . 
zone’ (shown in colors in Figure|7]i. Otherwise, x,i = 0. 


.(”)■ 

,..., 


, where Xd = 1 if the d-th sensor detects 


W}. For the n -th instance we 
an object inside its ‘detection 


5.2.1. Sensor Model 

Consider, for simplicity, a specific instance y = {yi,j)Yj =l (i n order to avoid here the use of the super index n). Moreover, let 
us denote as S 2 ,d] the position of the d-th sensor, and Zd the triangle of vertices s d, li = [2.5, 0] and I 2 = [7.5, 0] (the 

corners of the light source). This triangle Zd is the “detection zone” of the d-th sensor. Now, we define the indicator variable 

— 2’^ ~ 2 ) ^ z d,i,j = 0 tf — — ,j — — ^ ^ Zd, 


Zd,i,j — 1 if 
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(a) Accuracy 


(b) Exact Match 


(c) Hamming Score 



(d) Training Time (e) Test Time 

Fig. 6: Results of Nemenyi test, based on Table [ 7 ] and Table [8] If methods’ bars overlap, they can be considered statistically 
indifferent. The graphs based on time should be interpreted such that higher rank (more to the left) corresponds to slower (i.e., 
less desirable) times. 


where (i — \,j— y) is the middle point of the (i,j)- th pixel (tile), whose vertices are ( i,j ), (i — l,j — 1), (i — 1 ,j) and 
(i,j — 1) (for i,j = 2,..., W). Next, we define the variable 

w w 

Cd — ^ ' !Ji.j Zd.t.j ■ (10) 

i=l j=1 

which corresponds to the number of active tiles/pixels inside the triangle Zd associated to the d-th sensor. The likelihood 
function for the d-th sensor is then given by 


e 2 , 

P(x d = l|y) =p(x d = l\c d ) = < l-ei, 

[ 1 - eiexp[-0.1(c rf - 1)], 


c d = 0; 
c d = 1; 
c d > 1; 


( 11 ) 


and p(x d = 0|y) = 1 — p(x d = l|y); where ei = p(x d = 0|= 1) = 0.15 < 0.5 is the false negative rate and 
£2 = p{x d = l\c d = 0) = 0.01 < 0.5 is the false positive rate. 


5.2.2. Generation of Artificial Data 

Figure[7]shows a low dimensional scenario (L = 100, IT' = 10), for the purpose of a clear illustration, but we consider datasets 
with much higher levels of segmentation (namely LocaMOO, where L = 400, and LocallOk, where L = 10,000 - see Table [5]» 
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Fig. 7: An L = 10 x 10 = 100 tile localization scenario. D = 12 light sensors are arranged around the edges of the scenario 
at coordinates si, ..., s d , and there is a light source between points li = [2.5,0] and I2 = [7.5,0] (shown as a thick black line) 
on the horizontal axis. In this example, three observations are positive (Xd = 1). Note that the object in the bottom-right tile 
{yL = 1 in this case) cannot be detected. 


to compare the performance of several MOC techniques on this problem. Given a scenario with L = W x W tiles, D sensors 
and N observations, we generate the synthetic data, (x^, as follows: 

1. Start with an ‘empty’ y, i.e., y-ij = 0 for i,j = 1,..., W. 

2. Set y t j = 1 for relevant tiles to create a rectangle of width W/8 and height 2 starting from some random point y l r 

3. Create a 2 x 2 square in the corner furthest from the rectangle. 

4. Generate the observations according to Eq. GD- 

5. Add dynamic noise in y by flipping L /100 pixels uniformly at random. 

Any MLC method can be applied to this problem, to infer the binary vector y*, which encodes the presence of blocking- 
light elements in the room, given the vector x* of measurements from the light sensors. Finally, we also consider that each 
sensor provides M observations {xd,k}kLi C {0,1} M given the same y. 


5.2.3. Maximum A Posteriori (MAP) Estimator 


Given the likelihood function of Eq. (Ill, and considering a uniform prior over each variable c d , the posterior w.r.t. the <7-th 
triangle is 


p(c d \xd) cx p{x d \cd) = p(x d \y). 


If we also assume independency in the received measurements, the posterior density p(ci,..., Cd|x) can be expressed as 
follows 

D D D 

p(ci,...,c D |x)= JJp(c d |x) (X Y\_p( x d.\cd) = IJp(zd|y)- (12) 

d—1 d— 1 d—1 


We are interested in studying pi'yi.j |x) for 1 < i,j < W. but we can only compute the posterior distribution of the vari¬ 
ables {ci,. .. ci )}, which depend on y lt j through Eq. (10). Making inference directly on y,_ :i using the posterior distribution 
p{c\,. .. , c d |x) is not straightforward. Let us address the problem in two steps. First, the measurements received by each 
sensor, x d , can be considered as Bernoulli trials: if c d = 0, then Xd = 1 with probability = e 2 ; if c d > l, then Xd = 1 with 
























Algorithm 4 MAP inference using the sensor model 

input : {^d,k}^id=i ( measurements X { s d}^Li (sensor positions), li and I 2 (light source location). 

begin 

1. Initialize yij = 0.5 for i,j = 1,..., W. 

2. for d = 1,..., D; do 

(a) Calculate the detection triangle Z d . 

(b) If 0 d = Sfcli x d,k < 0.5, then set c d = 0. 

(c) Otherwise, if 6 d = SjtLi x d,k > 0.5, then set c d = 1. 

(d) If c d = 0, then set yij = 0 for all i,j G Z d . 

end 

3. For all d such that c d = 1 and for all i, j G Z d , check if the decision is still yij = 0.5. Then, set jji j = 1. 

4. The remaining tiles with yi j = 0.5 correspond to “shadow” zones, where we leave ynj =0.5. 

end 

output: jji j for i,j = 1,..., W. 


Table 9: Results using AlgorithmQwith D = 30 sensors. 


Measure 

W = 20 

w = too 

Accuracy 

0.523 

0.141 

Hamming score 

0.857 

0.795 

Times (total, s) 

1 

3 


success probability 9d = 1 — ei. Now, given M measurements for each sensor {xd,k}]f = i C {0, l} 717 and uniform prior density 
over 9d, the MAP estimator of 0,i is given by 

1 M 

= M ^2 Xd ^ k ’ 
k=1 

Then, if 9d < 0.5 we decide Cd = 0. Otherwise, if (),t > 0.5, we estimate c,i > 1. Considering a uniform prior over the pixels 
Uij, a simple procedure to estimate y from {ci,... c />} is the one described in Algorithm|4] 


5.2.4. Classifier Trellis vs MAP Estimator 

Results for CT are already given in Table[7](predictive performance) and Table[8](running time). Results in Table [Tjillustrate the 
robustness of the CT algorithm to address multi-output classification in several scenarios. Beyond the training set, no further 
knowledge about the underlying model is needed to achieve remarkable classification performance. To emphasize this property 
of CT, we now compare it to the MAP estimator presented above, which exploits a perfect knowledge of the sensor model. 

Table[9]shows the results using Algorithm [4] with D = 30 sensors and different values of W (i.e., the grid precision). The 
corresponding results obtained by CT are provided in Table 10 A detailed discussion of these results is provided at the end of 
the next Section. However, let us remark that increasing the number of tiles (i.e., W) for a given number of sensors D makes 
the problem harder, as a finer resolution is sought. This explains the decrease in performance seen in the tables as W increases. 


6. DISCUSSION 

As in most of the multi-label literature, we found that independent classifiers consistently under-perform, thus justifying the 
development of more complex methods to model label dependence. However, in contrary to what much of the multi-label 


Table 10: Results using CT (D = 30 sensors). 


Measure 

W = 20 

w = too 

Accuracy 

0.542 

0.133 

Hamming score 

0.969 

0.968 

Time (total, s) 

3 

7 














literature suggests, greater investments in modelling label dependence do not always correspond to greater returns. In fact, it 
appears that many methods from the literature have been over-engineered. Our small experiment in Table [2] suggests that none 
of the approaches we investigated were particularly dominant in their ability to uncover structure with respect to predictive 
performance. Indeed, our results indicate that none of the techniques is significantly better than another. Using ECC is a ‘safe 
bet’ in terms of high accuracy, since it models long term dependencies with a fully cascaded chain; also noted previously 
(e.g,. |20| |3l). In terms of EBCC (for which we elected to represent methods that uncover a structure), there was no clear 
advantage over the other methods, and surprisingly also no clear difference between searching for a structure based on marginal 
dependence versus conditional label dependence. This makes it more difficult to justify computationally complex expenditures 
for modelling dependence on the basis of improved accuracy; particularly so for large datasets, where the scalability is crucial. 

We presented a classifier trellis (CT) as an alternative to methods that model a full chain (as MCC or ECC) or methods 
that unravel the label graphical model structure from scratch, such as BCC. Our approach is systematic, we consider a fixed 
structure in which we place the labels in an ordered procedure according to easily computable mutual information measures, 
see Algorithm [T] An ensemble version of CT performs particularly well on exact match but, surprisingly, it does not perform 
much stronger than CT as we expected in the beginning. It does not perform as strong overall as ECC (although there is no 
statistically significant difference), but is much more scalable, as indicated in Table [4] 

The CT algorithm then emerges as a powerful MLC algorithm, able to excellent performance (specially in terms of average 
number of successfully classified labels) with near IC running times. Through the Nemenyi test, we have shown the statistical 
similitude between the classification outputs of (E ) CT and MCC/ECC, proving that our approach based on the classifier trellis 
captures the necessary inter-label dependencies to achieve high performance classification. Moreover, we have not analyzed 
yet the impact that the trellis structure chosen has in the CT performance. In future work, we intend to experiment with trellis 
structures with different degrees of connectedness. 

7. ACKNOWLEDGEMENTS 

This work was supported by the Aalto University AEF research programme; by the Spanish government’s (projects projects 
’COMONSENS’, id. CSD2008-00010, ’ALCIT’, id. TEC2012-38800-C03-01, ’DISSECT’, id. TEC2012-38058-C03-01); by 
Comunidad de Madrid in Spain (project ’CASI-CAM-CM’, id. S2013/ICE-2845); and by and by the ERC grant 239784 and 
AoF grant 251170. 


8. REFERENCES 

[1] R. Agrawal, T. Imielinski, and A. Swami. Mining association rules between sets in items in large databases. In Proc. of 
ACMSIGMOD 12, pages 207-216, 1993. 

[2] D. Barber. Bayesian Reasoning and Machine Learning. Cambridge University Press, 2012. 

[3] Weiwei Cheng, Krzysztof Dembczynski, and Eyke Hiillermeier. Bayes optimal multilabel classification via probabilistic 
classifier chains. In ICML ’10: 27th International Conference on Machine Learning, Haifa, Israel, June 2010. Omnipress. 

[4] Andrea Pohoreckyj Danyluk, Leon Bottou, and Michael L. Littman, editors. Proceedings of the 26th Annual International 
Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009, volume 382 of ACM 
International Conference Proceeding Series. ACM, 2009. 

[5] Krzysztof Dembczynski, Willem Waegeman, Weiwei Cheng, and Eyke Hiillermeier. On label dependence and loss mini¬ 
mization in multi-label classification. Mach. Learn., 88(l-2);5—45, July 2012. 

[6] Janez Demsar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 
7:1-30,2006. 

[7] Nadia Ghamrawi and Andrew McCallum. Collective multi-label classification. In CIKM ’05: 14th ACM international 
Conference on Information and Knowledge Management, pages 195-200, New York, NY, USA, 2005. ACM Press. 

[8] Anna Goldenberg and Andrew Moore. Tractable learning of large bayes net structures from sparse data. In Proceedings of 
the twenty-first international conference on Machine learning, ICML ’04, pages 44-, New York, NY, USA, 2004. ACM. 

[9] Yuhong Guo and Suicheng Gu. Multi-label classification using conditional dependency networks. In IJCAI ’ll: 24th 
International Conference on Artificial Intelligence, pages 1300-1305. IJCAI/AAAI, 2011. 


[10] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Reutemann Peter, and Ian H. Witten. The weka data 
mining software: An update. SIGKDD Explorations, 11(1), 2009. 

[11] Trevor Hastie and Robert Tibshirani. Classification by pairwise coupling. In Michael I. Jordan, Michael J. Kearns, and 
Sara A. Solla, editors. Advances in Neural Information Processing Systems , volume 10. MIT Press, 1998. 

[12] Markus Kalisch and Peter Biihlmann. Estimating high-dimensional directed acyclic graphs with the pc-algorithm. Journal 
of Machine Learning Research , 8:613-636, 2007. 

[13] Abhishek Kumar, Shankar Vembu, Aditya Krishna Menon, and Charles Elkan. Learning and inference in probabilistic 
classifier chains with beam search. In Peter A. Flach, Tijl De Bie, and Nello Cristianini, editors. Machine Learning and 
Knowledge Discovery in Databases, volume 7523, pages 665-680. Springer, 2012. 

[14] L. Ladick, C. Russell, P. Kohli, and P. H S Torr. Associative hierarchical crfs for object class image segmentation. In 
Computer Vision, 2009 IEEE 12th International Conference on, pages 739-746, Sept 2009. 

[15] Gjorgji Madjarov, Dragi Kocev, Dejan Gjorgjevikj, and Saso Deroski. An extensive experimental comparison of methods 
for multi-label learning. Pattern Recognition, 45(9):3084-3104, September 2012. 

[16] Jesse Read, Katrin Achutegui, and Joaquin Miguez. A distributed particle filter for nonlinear tracking in wireless sensor 
networks. Signal Processing, 98:121-134, 2014. 

[17] Jesse Read, Luca Martino, and David Luengo. Efficient monte carlo methods for multi-dimensional learning with classifier 
chains. Pattern Recognition, 47(3), 2014. 

[18] Jesse Read, Bernhard Pfahringer, and Geoff Holmes. Multi-label classification using ensembles of pruned sets. In 
ICDM’08: Eighth IEEE International Conference on Data Mining, pages 995-1000. IEEE, 2008. 

[19] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi-label classification. In ECML 
’09: 20th European Conference on Machine Learning, pages 254-269. Springer, 2009. 

[20] Jesse Read, Bernhard Pfahringer, Geoffrey Holmes, and Eibe Frank. Classifier chains for multi-label classification. Ma¬ 
chine Learning, 85(3):333-359, 2011. 

[21] Hongyu Su and Juho Rousu. Multilabel classification through random graph ensembles. In Asian Conference on Machine 
Learning (ACML), pages 404-418, 2013. 

[22] Grigorios Tsoumakas and loannis P. Vlahavas. Random k-labelsets: An ensemble method for multilabel classification. In 
ECML ’07: 18th European Conference on Machine Learning, pages 406^-17. Springer, 2007. 

[23] Celine Vens and Fabrizio Costa. Random forest based feature induction. In Proceedings of the 2011 IEEE 11th Interna¬ 
tional Conference on Data Mining, ICDM ’ 11, pages 744-753, Washington, DC, USA, 2011. IEEE Computer Society. 

[24] Raanan Yehezkel and Boaz Lerner. Bayesian network structure learning by recursive autonomy identification. Journal of 
Machine Learning Research, 10:1527-1570, 2009. 

[25] Julio H. Zaragoza, Luis Enrique Sucar, Eduardo F. Morales, Concha Bielza, and Pedro Larranaga. Bayesian chain classi¬ 
fiers for multidimensional classification. In 24th International Conference on Artificial Intelligence (IJCAI ’ll), 2011. 

[26] Min-Ling Zhang and Kun Zhang. Multi-label learning by exploiting label dependency. In KDD ’10: 16th ACM SIGKDD 
International conference on Knowledge Discovery and Data mining, pages 999-1008. ACM, 2010. 

[27] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and 
Data Engineering, 99(PrePrints):l, 2013. 



A. GRAPHICAL MODEL STRUCTURE LEARNING: FS VS. LEAD 


In order to delve deeper into the issue of structure learning, we generated a synthetic dataset, where the underlying structure 
is known. The synthetic generative model is as follows. For the feature vector, we consider a .D-dimensional independent 
Gaussian vector x £ M. D , where Xd ~ 7V(0,1) for d = 1,..., D. Let w i(l = 1,..., L) be a binary .D-dimensional vector 
containing exactly T ones (and thus D — T zeros), and let us assume that we have a directed acyclic graph between the labels 
in which each label has at most one parent. Both the vectors w/ and the dependency label graph are generated uniformly at 
random. Given the value of its parent label, y pa {e) £ {—1,1}, the following probabilistic model is used to generate the i-th 
label yf. 


+1, T 1 /2 w/x + €£ > 5; 
— 1, otherwise; 


(13) 


where <5 is a real constant and q ~ Af(a y pa (e),<x 2 ), with a£t and o £ R + . Note that, according to the model, yejy pa (e) is a 
Bernoulli random variable that takes value 1 with average probability 


P{yi = +%pa(f) 


+1) — Q 


& & y P a(£) 

\/l + cr 2 


where Q{x) = 1 — <i>(x) and <l>(x) is the cumulative distribution function of the normal Gaussian distribution. Consequently, 
with a and a 2 we control the likelihood of ye being equal to its parent y pa (e), thus modulating the complexity of inferring such 
dependencies by using the FS and LEAD methods. 

In Figure [8] we show three examples of synthetically generated datasets, in terms of their ground truth structure and the 
structure discovered using the FS and LEAD methods, for three different scenarios: ‘easy’ (a = 1, cr 2 = 1), ‘medium’ (a = 0.5, 
a 2 = 2), and ‘hard’ (a = 0.25, cr 2 = 5) datasets. Recall that we use a mutual information matrix for both methods, with the 
difference being that the LEAD matrix is based on the error frequencies rather than the label frequencies. Visually it appears 
that both FS and LEAD are able to discover the original structure, relative to the difficulty of the dataset. There appears to be 
a small improvement of FS over LEAD. This is confirmed in a batch analysis using the F-measure of 10 random datasets of 
random difficulty ranging between ‘easy’ and ‘hard’: FS gets 0.278 and LEAD gets 0.263. A more in depth comparison, taking 
into account varying numbers of labels and features, is left for future work. 


B. APPROXIMATE INFERENCE VIA MONTE CARLO 

A better understanding of the MLC/MOC approaches described in Section[3]and the novel scheme introduced in this work can 
be achieved by describing the Monte Carlo (MC) procedures used to perform approximate inference over the graphical models 
constructed to approximate p(y|x). 

Given a probabilistic model for the conditional distribution p(y|x) and a new test input x*, the goal of an MC scheme 
is generating samples from p(y|x*) that can be used to estimate its mode (which is the MAP estimator of y given x*), the 
marginal distribution per label (i.e., p(ye |x*) for £ = 1..., L) or any other relevant statistical function of the data. 


B.l. Bayesian networks 

In a directed acyclic graphical model, the probabilistic dependencies between variables are ordered. For instance, in the 
CC scheme p(y|x) factorizes according to Eq. If it is possible to draw samples directly from each conditional density 
p{ye\yi:£-h x ). then exact sampling can be performed in a simple manner. For i = 1,..., N s (where N s is the desired number 
of samples), repeat 


\ l ) 

y\ 

^ p(yi|x)), 

(i) 

y\ ■ 

~ p(y2\y ( i ] ,x-)), 

(i) 

Vl ' 



For Bayesian networks that are not fully-connected, as in BCC, the procedure is similar. Each sampled vector, yM = 
[Vi\ V 2 l \ • • ■ > \ for i = 1 ,..., N s , is obtained by drawing each individual component independently as y^ ~ p{ye ly^m) 

for l = 1, ..., L. 
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Fig. 8: Ground-truth graphs of the synthetic dataset (left) - easy, medium, and hard according to the table - and their recon¬ 
struction found by the FS strategy E) (middle) and LEAD strategy |[26l (right). 









B.2. Markov networks 


In an undirected graphical model (like that of a CDN), exact sampling is generally unfeasible. However, a Markov Chain Monte 
Carlo (MCMC) technique that is able to generate samples from the target density p(y|x) can be implemented. Within this 
class, Gibbs Sampling is often the most adequate approach. Let us assume that the conditional distribution p(y|x) factorizes 
according to an undirected graphical model as in Eq. |tJ. Then, from an initial configuration y(°) = ...., y^p], repeat for 

t = 1,... ,T: 

Vi ] ~ p{yAyi~ l \y { 2~ 1 \--->yiT 1) 
y? ~ p(y2\y { i ) ,y2~ 1 \-,yL~ 1) ^), 

y { L ~ p(yL\yi\y { 2 ) i*), 

where each label can be sampled by conditioning just on the neighbors in the graph, as seen from Eq. <0 Thus, 

yi ] ~p{ye\{y£ ) ■ iu e y ne (e),u< t},{y%T^ ■ y m e y ne{ t),m > f},x), 

which can be simply denoted as 't/p ~ p(yg |y„*!«)> y^m denoting the state of the neighbors of y£ at time t. 

Following this approach, the state of the chain yW = [y^ \ ..., y f^\ can be considered a sample fromp(y|x) after a certain 
“burn-in” period, i.e., for t > T c . Thus, samples for t < T c are discarded, whereas samples for t > T c are used to perform the 
desired inference task. Two problems associated to MCMC schemes are the difficulty in determining exactly when the chain has 
converged and the correlation among the generated samples (unlike the schemes in Section|BT| which produce i.i.d. samples). 


